Repository navigation
Release note: to_cframe() keeps the stored chunks of containers with a postfilter - #778
Open
FabianIsensee wants to merge 1 commit into
Open
FabianIsensee wants to merge 1 commit into
FabianIsensee wants to merge 1 commit into
Conversation
…stfilter Blosc/c-blosc2#858 makes blosc2_schunk_to_buffer() and friends write the chunks as they are stored for every storage. Before, a container with a postfilter that was not an in-memory contiguous frame was serialized with the postfilter's output, so SChunk.to_cframe() and NDArray.to_cframe() change for those containers once the bundled C-Blosc2 includes it. Release note and Notes sections in both docstrings, with how to serialize the postfiltered values instead. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
This branch has not been deployed
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Release note and
to_cframe()docstring notes for Blosc/c-blosc2#858. That PR changes whatSChunk.to_cframe()andNDArray.to_cframe()return for containers with a postfilter: the chunks as they are stored, for every storage. Today they return the postfilter's output unless the container is an in-memory contiguous frame, and in-memory containers are sparse by default. Francesc asked for a release note there; this is the python-blosc2 side.The text only becomes true once
BLOSC2_BUNDLED_VERSIONincludes c-blosc2#858 (it is at fa29efc0 now), so please merge this together with that bump or after it. Nothing else changes in python-blosc2.I checked it with python-blosc2 main built against the c-blosc2#858 branch, using a postfilter that adds 1000000 to an
int32arange, for in-memory and on-disk, contiguous and sparseSChunks andNDArrays:to_cframe()gives[0, 1]for all of them.SChunkfromschunk_from_cframe()reads[1000000, 1000001].NDArray.copy().to_cframe(), anSChunkmade froms[:], andblosc2.save()to a local path give[1000000, 1000001].Something I noticed while checking and did not change here:
NDArray.save()to a local path goes throughcopy(), so it stores the postfiltered values. To an fsspec URL without extra kwargs, it writesself.to_cframe(), which with c-blosc2#858 stores the chunks as stored. This already differed before #858 for in-memory contiguous arrays. I found this by reading the code; I could not run the fsspec path here. Should the two agree? I can send a follow-up either way.I used Claude Code to write the note and check it against a build; I reviewed the result.
🤖 Generated with Claude Code